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DESCRIPTION 
Picture Generating Apparatus and Picture Generating Method 

Technical Field 

This invention relates to a picture generatmg apparatus and a picture generating 
method for generating distance picture and variable-density (gradation) picture by 
employing the stereo method, and relates to a picture generating apparatus and a 
picture generating method for generating distance picture indicating distance between 
virtual position and object to be imaged and variable-density (gradation) picture when 
object to be imaged is imaged from virtual position by using two cameras or more. 

Background Art 

As an apparatus using the technique generally called stereo method, there is 
known, e.g., distance measurement unit (apparatus) for generating distance 
information from the camera unit up to object to be imaged. In this distance 
measurement unit, in order to measure three-dimensional coordinate position of the 
surface of object to be imaged, i.e., three dimensional shape thereof, pictures obtained 
by imaging, at the same time, the same object to be imaged by plural camera units 
having different view points are used to determine correspondence points 
corresponding to each other every respective pixels between those pictures to obtain 
distance information from the distance measurement unit up to the object to be 
imaged by that parallax. 

In the prior art, there will be explained the representative first technique for 
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generating distance picture consisting of distance infonnation and variable-density 
picture consisting of luminance information, etc. by camera unit disposed at virtual 
position and in virtual direction. At the time of generating distance picture and 
variable-density picture, three-dimensional measurement is first carried out by 
imaging object to be imaged by means of plural camera units. Then, spatial 
coordinates (shape) of object to be imaged are determined. Then, an approach is 
employed to virtually dispose camera unit within that space to generate, by using 
spatial coordinates (shape) of the object to be imaged, distance picture to be observed 
by the camera unit virtually disposed, and to generate variable-density picture. 

Explanation will now be given in connection with the second technique in 
which the stereo method is employed to generally determine distance picture with 
respect to object to be imaged. In this case, explanation will be given in connection 
with the example where, for the brevity of explanation, reference camera and 
detection camera for detecting distance between the reference camera and object to 
be imaged are used to determine correspondence points to generate distance picture. 

First, such an approach is employed to image (pick up image of) object to be 
imaged by reference camera and detection camera disposed at positions different &om 
each other to extract a predetermined area from picture data obtained by the reference 
camera to successively shift picture data of the extracted predetermined area on 
picture of the detection camera. At this time, the picture of the predetermined area 
is shifted on line called epipolar line determined by internal parameters and/or 



positional relationship of the detection camera. By successively shifting the picture 
of predetermined area on picture of the detection camera in this way, picture of the 
predetermined area imaged by the reference camera and picture imaged by the 
detection camera are compared with each other to determine degree of 
correspondence. Further, displacement quantity at displacement position where 
degree of correspondence is the highest is set as parallax at central pixel of the picture 
of the extracted predetermined area. In addition, by repeating these processing with 
respect to respective pixels of picture of the reference camera, distance pictures are 
generated. 

Namely, as shown in FIG. 1, object to be imaged is observed by reference 
camera and detection camera. When it is recognized that point P within the 
three-dimensional picture is observed at image pick-up point nt by the reference 
camera, and is observed at image pick-up point n^ by the detection camera, it is 
possible to determine three-dimensional position of the point P. In this case, it is not 
easy to judge (discriminate) that the corresponding point is image pick-up point of 
the detection camera corresponding to image pick-up point n^, of the reference 
camera. This is called correspondence point problem in the stereoscopic view. 

At the time of retrieving (searching) correspondence point generally carried 
out, image pick-up point n^ exists, as is clear with reference to FIG. 1, on line where 
plane determined by line of sight of the reference camera and angle of visibility of the 
detection camera and image pick-up plane (surface) of the detection camera intersect 



(cross) with each other. This line is called epipolar line. Further, when positional 
relationship between the reference camera and the detection camera and parameters 
inherent in respective cameras (focal distance or length, etc.) are known, it is possible 
to determine epipolar line on picture surface of detection camera every respective 
image pick-up points nt of the reference camera. It is thus sufficient to caiTy out 
retrieval of correspondence points on this epipolar line. 

An example where correspondence point of image pick-up point on picture 
imaged by the reference camera is detected on picture imaged by the detection camera 
will now be described. At this time, as shown in FIG. 2, with small area 100 around 
image pick-up point nt of the reference camera being as template, correlation 
(correlative) values are determined at several points on the epipolar line of picture of 
the detection camera. In this case, resolution of the epipolar line, i.e., resolution of 
distance are six points of image pick-up points n^, to n^s, and those image pick-up 
points n^, to n^e correspond to distance numbers 1 to 6 corresponding to distances 
from the reference camera. Further, these distance numbers 1 to 6 correspond to 
distances on the line of sight from the reference camera. When I(x) is assumed to be 
luminance value of picture imaged by reference camera and r(x') is assumed to be 
luminance value of picture imaged by detection camera, correlation is calculated by 
the following formula (1). 

Y.\i{x+i)-r{x^+i)\ -(1) 
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In accordance with the formula (1), according as correlation becomes larger 
(higher), degree of correspondence between picture imaged by the reference camera 
and picture imaged by the detection camera becomes higher, and according as 
correlation becomes smaller (lower), the degree of correspondence becomes lower. 
In addition, parameter corresponding to the degree of correspondence is assumed to 
be evaluation value. The relationship between the evaluation value and respective 
image pick-up points n,,, to n^^ on the epipolar line is shown in FIG. 3. In this case, 
according as correlation becomes smaller (lower), the evaluation value becomes 
larger, and according as correlation becomes larger (higher), the evaluation value 
becomes smaller. 

In accordance with this FIG. 3, image pick-up point n^ corresponding to the 
point in which evaluation value based on the correlation calculated by the formula (1 ) 
is the lowest (minimum) is assumed to be correspondence point. In this case, the 
distance number is "3". Alternatively, while distance corresponding to the point 
where evaluation value is the minimum may be determined from respective image 
pick-up points n^ as previously described, there are also instances where interpolation 
between sampled data is carried out from values at the periphery where the evaluation 
value is minimum to determine mmimum value. By determining distance of picture 
that the reference camera has imaged from the reference camera and the detection 
camera in this way, distance picture is generated along with variable-density picture 
having luminance information. 



6 

However, in the first technique for generating distance picture consisting of 
distance information and variable-density picture consisting of luminance 
information, etc. by camera unit disposed at the above-described virtual position and 
in virtual direction, since such an approach is employed to develop (expand) object 
to be imaged into three-dimensional picture to generate distance pictures by using 
three-dimensional pictures obtained by carrying out developing (expansion) in a 
three-dimensional manner, vast amount of calculations are required. Moreover, in 
this first technique, when three-dimensional measurement is carried out with respect 
to object to be imaged which has been imaged by plural camera units, it is necessary 
to image all coordinates which can be observed from the virtual camera unit, and 
three-dimensional measurements must be thus carried out from many view points. 
Further, in the first technique, it is also necessary to paste together pictures imaged 
by respective camera units when three-dimensional picture is generated. 

In addition, in the second technique which has been explained with reference 
to FIGS. 1 to 3, since such an approach is employed to shift picture imaged by the 
detection camera with picture imaged by the reference camera being as template to 
retrieve picture to thereby generate distance picture, existence of the reference camera 
is indispensable. 

Disclosure of the Invention 
This invention has been made in view of actual circumstances as described 
above, and its object is to provide a picture generating apparatus and a picture 



generating method capable of generating/by only detection camer^^ithout using 
/efelince earner^-, distance picture indicating~aiStance-from'the position where the 
reference camera is primarily disposed up to object to be imaged and variable-density 
(gradation) picture when object to be imaged is imaged from the reference camera. 

In order to attain the object as described above, a picture generating apparatus 
according to this invention comprises: two image pick-up means or more adapted 
for imaging (picking up image of) object to be imaged to generate picture data and 
respectively disposed at different positions; correlation detecting means for 
comparing, with each other, on epipolar line determined by connecting 
correspondence points of line of sight connecting virtual position and the object to be 
imaged and line of sight connecting positions of each of the imaging means and the 
object to be imaged, respective picture data generated by the respective image pick-up 
means to detect correlation therebetween; and distance picture generating means for 
generating distance picture indicating distance between the virtual position and the 
object to be imaged on the basis of the correlation detected by the correlation 
detecting means. 

In accordance with such a picture generating apparatus, picture data generated 
by two image pick-up means or more are used to compare, at the correlation detecting 
means, picture data generated by the respective image pick-up means to generate 
distance picture indicating distance between virtual position and object to be imaged 
at the distance picture generating means. 
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A picture generating method according to this invention comprises: imaging 
(picking up image of) object to be imaged by two (solid-state) image pick-up means 
(devices) or more respectively disposed at different positions to generate picture data; 
comparing, with each other, on epipolar line determined by connecting 
correspondence points of line of sight connecting virtual position and the object to be 
imaged and line of sight connecting positions of each of the image pick-up means and 
the object to be imaged, respective picture data generated by the respective image 
pick-up means to detect correlation therebetween; and generating distance picture 
indicating distance between virtual position and the object to be imaged on the basis 
of the detected correlation. 

In accordance with such picmre generating method, such a procedure is taken 
to compare picture data generated by two (solid-state) image pick-up (imaging) means 
(devices) or more to detect correlation therebetween to generate distance picture 
indicating distance between virtual position and object to be imaged on the basis of 
the correlation. 

Still further objects of this invention and more practical merits obtained by this 
invention will be more apparent from the embodiments which will be described 
below. 

Brief Description of the Drawings 
FIG. 1 is a view for explaining generation of distance information indicating 
distance between reference camera and object to be imaged by using reference camera 
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and detection camera by the conventional technique. 

FIG. 2A is a view for explaining that small area around image pick-up point 
of reference camera is caused to undergo template and FIG. 2B is a view for 
explaining determination of correlation values by several points on epipolar line set 
on picture (on screen) of detection camera. 

FIG. 3 is a view showing the relationship between evaluation values and 
positions at several points on epipolar line. 

FIG. 4 is a block diagram showing an example of a picture generating 
apparatus according to this invention. 

FIG. 5 is a block diagram showing an example of calibration unit for carrying 
out calibration on the premise that distance picture and variable-density picture are 
generated at virtual point A. 

FIG. 6 is a view for explaining an example for determining epipolar lines on 
respective detection pictures corresponding to respective coordinates of reference 
picture. 

FIG. 7 is a view showing observation points n^, and ri^ observed at the picture 
surface of detection camera in the case where object to be imaged exists at distances 
Z, and Zj. 

FIG. 8 is a view for explaining an example of calibration carried out on the 
premise that distance picture and variable-density picture are generated at distance 
picture generating section in this invention. 
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FIG. 9 is a view for explaining an example of look-up Table generated by the 
calibration unit. 

FIG. 1 0 is a flowchart showing an example of processing when distance picture 
and variable-density picture are generated at distance picture generating section 
provided in the picture generating apparatus according to this invention. 

FIG. 11 is a view for explaining derivation of pixels (n^, to n^^) every distance 
number on epipolar lines of detection pictures corresponding to coordinate n^ on 
picture that virtual camera should image. 

FIG. 12 is a view showing the relationship between evaluation value and 

distance nimiber. 

FIG. 13 is a flowchart showing another example of processing when distance 
picture and variable-density picture are generated at distance picture generating 
section provided in the picture generating apparatus according to this invention. 

FIG. 14 is a flowchart showing an example of processing when evaluation 
value picture is generated at distance picture generating section provided in the 
picture generating apparatus according to this invention. 

Best Mode For Carrying Out the Invention 

Explanation will be given below in detail with reference to the attached 
drawings in connection with an embodiment of this invention. 

This invention is applied to a picture generating apparatus (unit) 1 constituted 
as shown in FIG. 4, for example, and used when distance is measured to generate 
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distance picture. This picture generating apparatus 1 comprises plural detection 
cameras 3 (3a, 3b) for imaging (picking up image of) object 2 to be imaged, frame 
memories 4a, 4b into which picture data imaged by the plural detection cameras 3 are 
respectively stored, and a distance picture generating section 5 for generating distance 
picture and variable-density (gradation) picture by using picture data stored in the 
frame memories 4a, 4b. The picture generating unit (apparatus) 1 is adapted for 
imaging (picking up image of) the object 2 to be imaged by the detection cameras 3a, 
3b to thereby generate distance picture and variable-density picture (hereinafter 
referred to as "virtual picture") to be imaged by a virtual camera 3c which is virtually 
disposed at virtual point A and does not exist. 

The detection cameras 3a, 3b comprise, e.g., CCD image sensor, etc. and serve 
to pick up image of object 2 to be imaged to thereby generate picture consisting of 
luminance information, etc. (hereinafter referred to as "detection picture") to store 
data of the picture into the frame memories 4a, 4b. At this time, the detection cameras 
3 a, 3b respectively store the picture data thus generated into the frame memories 4a, 
4b every frame. 

Plural detection cameras 3 a, 3 b are disposed at positions different from the 
virtual point A in FIG. 4. In this example, detection cameras 3 are disposed at 
positions different from each other as the detection cameras 3a, 3b. It is to be noted 
that the positions where these detection cameras 3a, 3b are disposed and the number 
of cameras are not limited to the above, but they may be arbitrarily set. 
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The frame memories 4 (4a, 4b) respectively store thereinto picture data from 
the detection cameras 3a, 3b to output them to the distance picture generatmg section 
5. 

The distance picture generating section 5 processes picture data from the frame 
memories 4a, 4b to thereby generate virtual picture generated at virtual camera 3c. 

Explanation will now be given with reference to the calibration unit 1 0 of FIG. 
5 in connection with calibration carried out on the premise that distance picture and 
variable-density picture at virtual ^point A are generated at the distance picture 
generating section 5 provided in the above-described picture generating unit 1 shown 
in FIG. 4. It should be noted that, in the explanation of the calibration unit 10, the 
same reference numerals are respectively attached to the portions similar to those of 
the above-described picture generating unit 1 and their detailed explanation will be 
omitted. 

The calibration unit (apparatus) 10 shown in FIG. 5 comprises a reference 
camera 1 1 disposed at the above-described vutual point A, and a frame memory 12 
into which picture data imaged by the reference camera 1 1 is stored, whereby picture 
imaged by reference camera 1 1 (hereinafter referred to as "reference picture") in 
which object 2 to be imaged is imaged by a calibration section 1 3 and picture unaged 
by detection cameras 3a, 3b (hereinafter referred to as "detection picture") are used 
to carry out calibration. 

The calibration section 1 3 carries out processing to allow the detection cameras 
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3a, 3b and the reference camera 1 1 to undergo positioning to determine epipolar line 
on the detection picture to determine in advance correspondence relationship between 
coordinates of reference picture and detection picture to , e.g., store it as look-up table 
into the above-described distance picture generating section 5. 

This epipolar line is (straight) line where plane determined by optical centers 
(optical axes) of the reference camera 1 1 and the detection cameras 3a, 3b and 
observation point n^ of the reference camera 1 1 and (surface of) picture of each of the 
detection cameras 3a, 3b intersect (cross) with each other. Now, explanation will be 
given in connection with an example for determining epipolar line on each of 
detection pictures corresponding to each of coordinates of reference picture. It is here 
assumed that, for example, as shown in FIG. 6, point P on the plane placed within 
three-dimensional space is observed at point n^ on reference picture of the reference 
camera 11 and is observed at point n^ on detection picture of the detection camera 3. 
In the case where such a relationship holds, when projection (projective) transform 
matrix of 3x3 for carrying out transform processing from the point nb to the point 
is assumed to be H, the point n^ of the detection picture is expressed as follows: 
nd = H*nb •"(2) 

Further, this distance picture generating section 5 implements a predetermined 
projection transform processing to reference picture imaged by the reference camera 
11 while changing parameters of this projection transform matrix H to determine 
parameters of the projection transform matrix H so that error of luminance between 
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detection picture and reference picture becomes minimum. By determining projection 
transform matrix H, the calibration section 13 can retrieve, from on the detection 
picture, correspondence point corresponding to an arbitrary point of the reference 
picture. In this example, as a method of determining projection transform matrix H 
in which transformed detection picture and reference picture are caused to be 
precisely in correspondence with each other (picture matching method), 
Levenberg-Marquardt minimum method (L-M method) is used. This calibration 
section 13 determines, every distance, projection transform matrix H by using such 
a technique to determine epipolar line by using the projection transform matrix H. 

Namely, in this calibration section 13, as shown in FIG. 7, for example, when \ 
the above-described L-M method is used to determine projection transform matrix H, 
with respect to distance Z, from the reference position, point n,,, on detection picture 
corresponding to point nt on the reference picture at distance Z, from the reference 
position can be determined. In this case, since point n^, on the reference picture is 
arbitrary, the calibration section 13 can calculate, as point n^ on detection picture, / 
correspondence points corresponding to all points on reference picture by using the ^ 
projection transform matrix H, at distance Z,. In a manner similar to the above, the | 
calibration section 13 determines projection transform matrix H2 in the case where 
p lane to be obser ved is placed at distance Z2 so as to become in parallel to plane 
placed at position of distance Z,. ^ 

Further, the calibration section 1 3 calculates that the point nb on the reference 
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picture is projected onto respective points n,,, and n^j on the detection picture by 
projection transform matrix H, and projection transform matrix H2 corresponding to 
distance Z, and distance in the case where the point nt is located at distance Z, and 
Z2. As stated above, the calibration section 1 3 interpolates two points of point n^, and 
point on the detection picture to calculate ^pipolarl jne to thereby have ability to 
determine the relationship of the^^^on the^feference-picture, the distance Z and 

0 the point n^ on the detection picture. By storing such proj ection transform matrix H 

1 every distance Z, distance information n/^^ oictu^o be generated ^y virtual./ 
.camerTs^ovided at the picture generating unit 1 can be obtained by using the 

method which will be described later. It is to be noted that when interpolation is 
carried out in this way, it is necessary to set plane placed at distance positioned 
between the distances Z, and Z^. It is to be further noted that the detail relating to the 
example of calibration that the above-described calibration section 13 carries out is 
described in the Japanese Patent Application No. 20795 1/1 997. 

As a method of determining distance Z from point nb on the reference picture 
and point n^ on the detection picture, the following method may be employed. 
Namely, the calibration section 1 3 carries out calibration shown in FIGS. 6 and 7 on 
calculated epipolar line to thereby determine the relationship between point xv^ on 
reference picture and point n^, on detection picture to store this relationship, e.g., into 
look-up table. 

In this calibration, by using the fact that, e.g., as shown in FIG. 8, point at 
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which there intersect (cross) line of sight connecting point n^, on the epipolai:.line.ol 
detection picture and optical center and line of sight connecting (poiii^ 
iSS^^Tpi^re^and optical center of the reference camera 11 corresponds to 
distance Z, from the reference camera 1 1 up to object to be imaged, distance Z, is 
caused to be distance number "1". This epipolar line is calculated by usmg the 
above-described projection transform matrix H. Moreover, point where there are 
intersect (cross) line of sight connecting point n^j on the epipolar line of detection 
picture and optical center and line of sight connecting point n^ on the reference picture 
and optical center of reference camera 1 1 corresponds to distance Zj from the 
reference camera 1 1 up to the object to be imaged. At this time, distance is caused 
to be distance number "2". 

In this way, the calibration section 13 detects, as distance numbers 1 to 6, 
distances Z, to Z^ corresponding to the points where lines of sight connecting points 
nj, to n^e on epipolar lines of detection picture and line of sight connecting point ni, 
on the reference picture and optical center of the reference camera 1 1 intersect or 
cross with each other successively with respect to points nj, to of detection picture 
to thereby carry out calibration before distance picture generation carried out at the 
above-described picture generating apparatus 1. Further, the calibration section 13 
can obtain distance numbers 1 to 6 of pictures to be imaged at point n^ on the 
reference picture from the relationship between point n^ on the reference picture and 
point Ud on the detection picture. 
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In addition, the calibration section 13 may set respective planes with respect 
to respective distances Z, to Zg to generate respective projection transform matrices 
indicating correspondence relationship between point n^ on the reference picture and 
point nj on the detection picture, and, e.g., may set planes only with respect to three 
distances of distances Z„ Z3, Z^ to determine projection transform matrix indicating 
correspondence relationship between point nb on the detection picture and point on 
the detection picture thereafter to determme, by interpolation, projection transform 
matrix with respect to distances Z2, Z4 positioned between distances Z„ Z3, Zg. 

Namely, this calibration section 13 successively carries out, on the epipolar ' 
line, calibration of point on the detection picture corresponding to point n^, on the 
reference picture from the optical center (optical axis) to thereby determine 
correspondence points n^j to n^e on the epipolar line to thereby determine distances 
Z, to Zg to allow correspondence points n^, to n^g and distance numbers 1 to 6 to have 
correspondence relationship therebetween to determine the relationship between point 
n^ on the reference picture and point n^ on the detection picture to prepare look-up 
table as shown in FIG. 9, for example. 

In this FIG. 9, there are stored, e.g., points (n^,, n^z, n^^, -) on detection picture 
that detection camera 3a located at distances (Zj, Z2, Z3, •••) corresponding to a certain 
point n^ existing on the reference picture has imaged and points (nj,, n^z, nd3» "O on 
detection picture that detection camera 3b located thereat has imaged. Further, such 
look-up tables are generated, e.g., every respective pixels on reference pictures, and 



13 



18 

reference is made to pixels on the reference picture in the calibration section 13 as 



pixels on the^^'vi^l pictufe-.when distance picture is generated at the picture 
generating unit 1 . 

Explanation will now be given with reference to the flowchart of FIG. 10 in 
connection with the example where, at the distance picture generating section 5 
provided in the picture generating unit (apparatus) 1, picture data generated at 
detection cameras 3a, 3b are used to generate distance picture and variable-density 
picture by making reference to look-up table generated in the above-described 
calibration unit 10. 

In accordance with the flowchart shown in FIG. 10, initially, at step STl , the 
distance picture generating section 5 initializes coordinate i of pixel of picture data 
Qf(ySu al picture o f^ irtiiaJcajnera 3c)to thereby set i to zero (i=0). 

Subsequently, at step ST2, the distance picture generating section 5 initializes 
distance number j corresponding to resolution in the distance direction when 
calibration is carried out to set j to zero 0=0). Thus, the processing proceeds to step 
ST3 and step ST4. 

At the step ST3, as shown in FIG. 1 1, the distance picture generating section 
5 derives (generates) pixels n^, (i, j) (n^i to n^^) at distance number j on the epipolar 



line of detection picture corresponding to coordiante i (nt) on picture that tii e virtual 
camera 3c should image. Namely, when coordiante i on th^^ SpIHui-e --^at the 
<^i^^amem^c^ould image is determined at step STl, the distance picture 
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generating section 5 detects, from the look-up table, pixel n^^ (i, j) on the epipolar line 
of the detection camera 3 a determined by the above-described processing. Thus, the 
processing proceeds to step ST5. It is to be noted that position of determined 
(detected) pixel n^g (i, j) of detection picture may be represented by sub-pixels in 
addition to integer value. 

At the step ST5, the distance picture generating section 5 extracts, as small area 
WA (i, j), the portion in the vicinity of pixel n^^ (i, j) at distance number j determined 
at the above-described step ST3 with the pixel n^^ (i, j) being as center. 

On the other hand, the distance picture generating section 5 carries out, at step 
ST4, also with respect to detection camera 3b, the processing which has been 
explained at the above-described step ST3. Namely, this distance picture generating 
section 5 detects, from the look-up table, pixel n^b (i, j) at distance number j on 
epipolar line on detection picture that detection camera 3b has imaged corresponding 
to coordinate i on picture that the virtual camera 3 c should image. Thus, the 
processing proceeds to step ST6. 

Also at step ST6, the distance picture generating section 5 carries out 
processing similar to the above-described step ST5 with respect to the detection 
camera 3b. Namely, the distance picture generating section 5 extracts, as small area 
WB (i. j), the portion in the vicinity of pixel (i, j) at distance number j determined 
at the above-described step ST4 with the pixel n^b (i, j) being as center. 

Then, at step ST7, the distance picture generating section 5 compares small 



area 
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WA (ij) extracted at the above-described step ST5 and small area WB (ij) 
extracted at the step ST6 to thereby calculate correlation to obtain evaluation value 
sG). Here, the evaluation value sQ) indicates correlation at the distance number j, and 
is calculated, e.g., by the following formula. 

2;i/u+o-/'U'+oi -(3) 



According as correlation becomes higher, it is indicated that (the degreee of) 
analogousness between small area WA (i.j) and small area WB (i J) becomes higher. 
Moreover, as the result of the fact that the evaluation value s(j) is calculated by the 
above-mentioned formula (2) in correspondence with correlation at the distance 
picture generating section 5, according as correlation becomes higher, the evaluation 
value has smaller value. Namely, this distance picture generating section 5 compares 
luminance pattern of small area WA (iJ) of detection picture generated by the 
detection camera 3a and that of small area WB (ij) of detection picture generated by 
the detection camera 3b to thereby generate (the degree of) analogousness as 

evaluation value s(j). 

Then, at step ST8, the distance picture generating section 5 recognizes distance 
number j to thereby judge whether or not evaluation values sG) have been calculated 
with respect to all distance numbers. When it is judged that the section 5 has carried 
out processing with respect to all distance numbers j, the processing proceeds to step 
STIO. On the other hand, when it is judged that the section 5 has not yet carried out 
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processing with respect to all distance numbers j, the processing proceeds to step ST9 
to increment distance number j. Thus, the processing proceeds to steps ST3 and ST4. 
Namely, the distance picture generating section 5 carries out the processing explained 
at the above-described steps ST3 to ST7 in connection with all distance numbers j 
with respect to coordinates i of pixels of virtual picture of virtual camera 3c. 

Then, at step ST 10, the distance picture generating section 5 selects evaluation 
value sO„,i„) having minimum value of respective evaluation values sQ) in connection 
with all distance numbers j with respect to coordinate i of virtual picture. When 
evaluation values sG) corresponding to distance numbers 1 to 6 are obtained at the 
distance picture generating section 5, as shown in FIG. 12, for example, evaluation 
value sGmin) having minimum value when distance number is "3" is obtained. At this 
time, the distance picture generating section 5 may interpolate between respective 
evaluation values s(j) obtained at the step ST7 to determine minimum evaluation 
value sCimin). The distance picture generating section 5 is operative so that when 
interpolation is made to obtain minimum evaluation value sGmin)> the value in which 
the distance number n^^ is "3,3" is obtained as minimum evaluation value s(jmin)- 
Accordingly, this distance picture generating section 5 obtains evaluation value s(jmin) 
by interpolation to thereby permit accuracy of evaluation value to be high. 

Then, at step STll , the distance picture generating section 5 stores, as distance 
information of coordinate i of distance picture imaged by the virtual camera 3 c, 
distance Z corresponding to distance number j that the evaluation value s(jmin) having 
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minimum value obtained at the above-described step ST 10 indicates. 

Then, at step ST 12, the distance picture generating section 5 stores, as 
luminance information at coordinate i of virtual picture, e.g., one value or both values 
of pixels of detection picture imaged by the detection camera 3a and detection picture 
imaged by the detection camera 3b corresponding to distance number j that evaluation 
value sOmJ having minimum value obtained at the above-described step STIO 
indicates. 

Then, at step ST13, the distance picture generating section 5 carries out the 
above-described processing of steps ST3 to ST12 with respect to respective 
coordinates i on the virtual camera 3c to thereby judge whether or not processing for 
determining distance information and luminance information have been made with 
respect to all coordinates i of the virtual picture. Further, when the distance picture 
generating section 5 has judged that distance information and luminance information 
are obtained with respect to all coordinates i, it completes the processing. On the 
other hand, when the section 5 has judged that distance information and luminance 
information are not determined with respect to all coordinates i, it increments value 
of coordinate i at step ST14. Thus, the processing returns to the step ST2 to carry out 
processing of steps ST2 to ST12 with respect to the incremented coordiante i+1 to 
repeat processing until distance information and luminance information are 
determined with respect to all coordinates i. 

The picture generating unit 1 provided with such distance picture generating 
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section 5 can generate distance picture indicating distance from viitual point A where 
camera is primarily disposed up to object 2 to be imaged and variable-density picture 
when the object 2 to be imaged is imaged from the vii-tual point A by only detection 
cameras 3a, 3b without disposing camera at virtual position with respect to virtual 
picture to be generated at the virtual camera 3c. Accordingly, in accordance with this 
picture generating unit 1, it is possible to generate distance picture having distance 
information with respect to respective pixels which cannot be electronically generated 
as picture only by the virtual camera 3 c. 

Moreover, in accordance with the picture generating unit 1, when distance 
picture and variable-density picture from virtual camera disposed at virtual position 
are generated, camera is actually located (disposed) at the position from the virtual 
camera to carry out calibration, thereby making it possible to generate distance picture 
and variable-density picture to be generated by the virtual camera without necessity 
to carry out complicated processing such that expansion is made into the coordinate 
system of the three-dimensional space of object to be imaged as in the prior art. 

Further, in accordance with this picture generating unit (apparatus) 1, in the 
case where, e.g., there exists screen for displaying picture at the front of user, even 
in the state where camera cannot be disposed at the front of user, plural detection 
cameras 3 are disposed at the periphery of user, whereby image pick-up operation is 
made from the front of user, thus making it possible to generate distance picture and 
variable-density picture to be generated. 
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Furthermore, in accordance with this picture generating unit (apparatus) 1, 
there may be disposed, at position of the virtual camera 3c, image pick-up camera of 
high fineness or camera of the type different from detection cameras 3a, 3b like 
camera using film, etc. Such picture generating unit 1 can pick up image of 
variable-density picture consisting of luminance infoniiation and color information 
independently of processing for measuring distance between virtual camera 3c 
disposed at virtual position and object 2 to be imaged, and can also measure distance 
of object 2 to be imaged which has been imaged by ordinary camera disposed at 
virtual position where virtual camera 3c is disposed. 

Explanation will now be given with reference to the flowchart shown in FIG. 
13 in connection with another example where the distance picture generating section 
5 generates distance picture and variable-density picture by using picture data 
generated by detection cameras 3a, 3b. 

Initially, at step ST21, this distance picture generating section 5 initializes 
distance number j correspondmg to resolution in distance direction when calibration 
is carried out to set j to zero (j=0). 

Then, at step ST22, the distance picture generating section 5 initializes 
coordinates i of respective pixels on virtual picture of virtual camera 3 c to set i to zero 
(i=0). Thus, the processing proceeds to step ST23 and step ST24. 

At the step ST23, the distance picture generating section 5 derives pixel nj^i, 
j) at distance number j on epipolar line on detection picture that detection camera 3 a 



ru 



25 

has imaged corresponding to coordinate i on virtual picture that viitual camera 3c 
should image similarly to the above-described step ST3. Thus, the processing 
proceeds to step ST2 5. 

At the step ST25, the distance picture generatmg section 5 stores, as pixel Da(i) 
of parallax picture Da, lummance value of pixel n^/i, j) derived at the 
above-described step ST23. 

On the other hand, at the step ST24, the distance picture generating section 5 
w derives, similarly to the above-described step ST4, pixel nab(i, j) at distance number 
j on epipolar line on detection picture corresponding to coordinate i on virtual picture 
that virtual camera 3c should image. Thus, the processing proceeds to step ST26. 

Further, at the step ST26, the distance picture generating section 5 stores, as 
pixel Db(i) of parallax picture Db, luminance value of pixel ndb(i, j) derived at the 
above-described step ST24. 

As stated above, the distance picture generating section 5 carries out processing 
indicated by step S25 and step ST26 to thereby store pixels Da(i), Db(i) of parallax 
pictures Da, Db at distance number j of respective detection cameras 3a, 3b at 
coordiante i=0 of virtual picture. At this time, the distance picture generating section 
5 derives, by making reference to the look-up table, pixels Da(i), Db(i) corresponding 
to distance number j when areas on epipolar lines of respective detection pictures are 
calibrated by the calibration unit 10. 

Then, at step ST27, the distance picture generating section 5 judges whether 
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or not pixels Da(i), Db(i) of parallax pictures Da, Db are stored with respect to all 
coordinates i of viitual picture at distance number j. Further, when the distance 
picture generating section 5 has judged that pixels Da(i), Db(i) of parallax pictures 
Da, Db are stored with respect to all coordinates i, the processing proceeds to step 
ST29. On the other hand, when the section 5 has judged that such pixels are not 
stored with respect to all coordiantes i, the coordiante i is incremented at step ST28 
so that coordinate i+1 resuh. Thus, the processing returns to the step ST23 and the 
step ST24 for a second time. Namely, the distance picture generating section 5 
repeats processing of step ST23 to step ST28 until pixels Da(i), Db(i) of parallax 
pictures Da, Db are stored with respect to all coordinates i at distance number j. 

At step ST29, the distance picture generating section 5 compares, by using 
above-mentioned formula (1), pixel Da(i, j) included in parallax picture Da and pixel 
Db(i, j) included in parallax picture Db which are obtained by carrying out the 
above-described processing of steps ST23 to ST28 to thereby determine correlation 
between detection picture imaged by the detection camera 3 a and detection picture 
imaged by the detection camera 3b to detennine evaluation value s(i, j) corresponding 
to correlation with respect to each pixel. Further, the distance picture generating 
section 5 generates evaluation value picture S(i, j) consisting of evaluation value s(i, 
j) of coordinate i at distance j. It is to be noted that the detail with respect to 
processing for generating this evaluation value picture S(i, j) will be described later. 

Then, at step ST30, the distance picture generating section 5 recognizes 
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distance number j to thereby judge whether or not evaluation value pictures S(i, j) 
have been generated with respect to all distance numbers j. When processing have 
been carried out with respect to all distance numbers j, the processing proceeds to step 
ST32. On the other hand, when processing have not been carried out with respect to 
all distance numbers j, the processing proceeds to step ST31 to increment distance 
number j to allow the distance number to be j+1 to repeat processing of steps ST23 
to ST3 1 for a second time. 

At the step ST32, coordinates i of respective pixels on virtual picture of the 
virtual camera 3c are initialized so that coordinate i=0 resuhs. 

Then, at step ST33, the distance picture generating section 5 retrieves value 
where evaluation value s(j) which is value of evaluation value picture S(i, j) every 
distance number j at coordiante i obtained by the above-described processing is 
minimum to derive (generate) distance number j^^n corresponding to pixel s(i, j„i„) 
having minimum evaluation value. In addition, this distance picture generating 
section 5 may interpolate respective evaluation values s(j) every respective distance 
numbers j to determine minimum evaluation value sQ^i„). As stated above, the 
distance picture generating section 5 determines, by interpolation, minimum 
evaluation value sQ^J, thereby permitting accuracy of the evaluation value s(j) to be 
higher. 

Then, at step ST34, the distance picture generating section 5 stores, as distance 
information of coordinate i of distance picture imaged by the virtual camera 3C, 
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distance Z corresponding to distance number j,„i„ that the minimum evaluation value 
s(i Jmin) obtained at the above-described step ST33 indicates. 

Then, at step S3 5, the distance picture generating section 5 stores, as luminance 
information at position i of virtual picture, e.g., one value or average (mean) value of 
both values of luminance information of pixels of detection picture imaged by 
detection camera 3a and detection picture imaged by detection camera 3b 
corresponding to distance number j,„i„ that the minimum evaluation value s(i, j^m) 
obtained at the above-described step ST33 indicates. 

Then, at step ST36, the distance picture generating section 5 caiTies out, with 
respect to respective positions i of pixels on the virtual camera 3c, the 
above-described processing of step ST33 to step ST35 to thereby judge whether or 
not processing for determining distance information and luminance information are 
carried out with respect to all coordinates i of virtual picture. Further, when it is 
judged that distance information and luminance information are determined with 
respect to all coordinates i, the distance picture generating section 5 completes the 
processing. On the other hand, when it is judged that distance information and 
luminance information are not determined with respect to all coordinates i, the 
coordinate i is incremented at step ST37. Thereafter, the processing returns to the 
step ST 33. Thus, the distance picture generating section 5 carries out processing of 
step ST33 to step ST36 with respect to the incremented coordinate i+1 to repeat 
processing of steps ST33 to step ST37 until distance information and luminance 
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information have been determined with respect to all coordinates i. 

An example for generating evaluation value picture S(i, j) from parallax picture 
Da and parallax picture Db at the above-described step ST29 will now be described 
with reference to FIG. 14. 

At the time of generating evaluation value picture S(i, j), the distance picture 
generating section 5 initializes, initially, at step ST41, coordinate i of picture of 
evaluation value picture S(i, j) into zero (0). 

Then, at step ST42, the distance picture generating section 5 compares 
luminance patterns of small area in the vicinity of parallax picture Da(i) and small 
area in the vicmity of parallax picture Db(i) to thereby calculate evaluation value s(i) 
every small area. As a result, there exists evaluation value sG) corresponding to 
distance number j as information of each pixel. At this time, the distance picture 
generating section 5 may calculate evaluation value s(j) by using the 
above-mentioned formula (1). 

Then, at step ST43, the distance picture generating section 5 stores, as pixels 
of evaluation value picture S(i, j), evaluation values obtained by calculation at the 
above-described step ST42. 

Then, at step ST44, the distance picture generating section 5 judges whether 
or not pixels of evaluation value picture S(i, j) are generated with respect to the 
entirety of picture on screen. When it is judged that storage has been carried out as 
pixels of evaluation value picture S(i, j) with respect to all pixels from coordinate i 
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of virtual picture, the processing is completed. Wlien it is judged that storage has not 
yet been carried out as pixels of evaluation value picture S(i, j) with respect to all 
pixels from coordinate i, the coordinate i is incremented into coordinate i+1. The 
section 5 carries out processing of step ST42 and step ST43 for a second time to 
repeat processing until pixels of evaluation value picture S(i, j) are stored resultantly 
with respect to all coordinates i. 

As stated above, the picture generating unit 1 can generate distance picture 
indicating distance from the position where camera is primarily disposed until object 
2 to be imaged and variable-density picture when the object 2 to be imaged is imaged 
from virtual camera 3c by only detection cameras 3a, 3b without disposing camera 
at virtual position with respect to virtual picture to be generated at virtual camera 3c. 

Further, this picture generating unit (apparatus) 1 compares luminance patterns 
of small area in the vicinity of parallax picture Da(i) and small area in the vicinity of 
parallax picture Db(i) to thereby calculate evaluation values sG) every small areas to 
calculate coordinates of virtual picture while successively incrementing such 
coordinate to thereby generate evaluation value picture S(i, j) every respective 
distance numbers to select minimum evaluation value sQ^J to generate distance 
picture consisting of distance information and variable-density picture consisting of 
luminance information. Thus, quantity of calculations can be reduced as compared 
to the above-described processing shown in FIG. 10 or FIG. 12. 

It is to be noted that while, in the processing of the distance picture generating 
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section 5 which has been explained with reference to the above-described flowcharts, 
the example for determining distance information and luminance information with 
respect to coordiantes i of the entirety of virtual picture has been described, distance 
information and luminance information may be determined with respect to a portion 
of coordinates i of virtual picture. 

Moreover, at the time of calibration carried out on the premise of the 
processing of the distance picture generating section 5 which has been explained with 
reference to the above-described flowcharts, such an approach may be employed to, 
e.g., project pattern light onto the plane to be imaged to thereby paste texture onto the 
object to be imaged. As stated above, the picture generating unit 1 pastes texture onto 
object to be imaged at the time of calibration, thereby making it possible to improve 
accuracy of calibration to improve accuracy of distance measurement. 

In addition, in order to improve accuracy of distance measurement, this picture 
generating unit (apparatus) may comprise: camera for projecting, e.g., pattern light 
of infrared area onto object to be imaged, disposed at position where virtual camera 
3 c is disposed and including infrared ray shielding filter; and detection cameras 3 
positioned therearound. At this time, camera in which calibration has been carried 
out is used as it is as virtual camera. Thus, this picture generating unit (apparatus) 1 
receives pattern light of infrared area by the detection cameras disposed therearound 
to generate distance picture, and to generate variable-density picture by camera 
disposed at the position where virtual camera 3c is disposed. At this time, picture 
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generated by camera disposed at the position where vii-tual camera 3c is disposed is 
utilized as variable-density picture as it is. 

Industrial Applicability 
As explained above in detail, the picture generating apparatus according to this 
invention comprises: correlation detecting means for comparing, with each other, on 
epipolar line determined by connecting correspondence points of line of sight 
connecting virtual position and object to be imaged and line of sight connecting 
position of each image pick-up means and the obj ect to be unaged, respective picture 
data generated by the respective image pick-up means to detect correlation 
therebetween; and distance picture generating means for generating distance picture 
indicating distance between virtual position and object to be imaged on the basis of 
correlation detected by the correlation detecting means, whereby picture data 
generated by two image pick-up means or more are used to compare, at the 
correlation detecting means, picture data generated by the respective image pick-up 
means to detect correlation therebetween. Accordingly, it is possible to generate, 
without disposing camera at the virtual position, distance picture mdicating distance 
from the virtual position to the object to be imaged and variable-density picture when 
the object to be imaged is imaged from the virtual position. 

In addition, the picture generating method according to this invention 
comprises: comparing, with each other, on epipolar line determined by connecting 
conespondence points of line of sight connecting virtual position and object to be 
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imaged and line of sight connecting position of each image pick-up means and the 
object to be imaged, respective picture data generated by the respective image pick-up 
means to detect correlation therebetween. Accordingly, it is possible to generate, on 
the basis of the detected correlation, distance picture indicating distance from virtual 
position to object to be imaged and variable-density picture when the object to be 
imaged is imaged from the virtual position. 



